Super resolution infrared imaging system

ABSTRACT

A method for super resolution enhancement of infrared imaging data employing an image array having simultaneous, high speed, randomly addressable windows of pixels for small regions of interest (ROI) within a large imaging sensor field of view. Data is retrieved from the ROI window pixels at an increased data rate and the array is dithered to produce sub-pixel motion consistent with the increased data rate. A selected super resolution (SR) algorithm is then applied to the data retrieved from the ROT window pixels at the increased data rate. The reduced data set substantially relieves the computational load on the processor compared with the requirement of performing SR processing on the entire frame of image data. The output from the SR algorithm is either used to replace the LR frame data in the image or is displayed as a separate image to the user.

REFERENCE TO RELATED APPLICATIONS

This application claims the priority of U.S. Provisional ApplicationSer. No. 61410652 filed on Nov. 5, 2010 entitled SUPER RESOLUTIONINFRARED IMAGING SYSTEM BACKGROUND INFORMATION the disclosure of whichis incorporated herein by reference as though fully set forth.

BACKGROUND INFORMATION

1. Field

Embodiments of the disclosure relate generally to the field of read-outintegrated circuits (ROIC) for infrared sensors and more particularly toembodiments for an infrared imaging camera system that provides superresolution image data using a multiple windowing focal plane arrayreadout integrated circuit (ROIC) that permits a selected number ofsmall, two-dimensional image portions to be read off of the focal planearray (FPA) at high rates with sub-pixel motion of imagery for ditheringof the data in a super resolution processor.

2. Background

Infrared detection and imaging systems are being employed to sensetemperature differences to create scenes displaying various objects.Current imagers provide exceptional acuity for scene reproduction. Tomaintain desired frame rates lower resolution in output from the imagingarray is often required. However, certain features of interest in animage may require super resolution for adequate display. Currentresolution enhancement systems require operation on the pixels of theentire array significantly reducing frame rate capability for display.

It is therefore desirable to provide arrays with ROICs which allow superresolution processing on only selected portions of the array to maintainhigh frame rates for the overall image.

SUMMARY

Exemplary embodiments provide a method for super resolution enhancementof infrared imaging data employing an image array having simultaneous,high speed, randomly addressable windows of pixels for small regions ofinterest (ROI) within a large imaging sensor field of view. Data isretrieved from the ROI window pixels at an increased data rate and thearray is dithered to produce sub-pixel motion consistent with theincreased data rate. A selected super resolution (SR) algorithm is thenapplied to the data retrieved from the ROI window pixels at theincreased data rate.

In exemplary embodiments, the output from the super resolution (SR)algorithm is either used to replace the low resolution (LR) frame datain the image or is displayed as a separate image to the user. Ditheringmay be accomplished with mechanical or optical dithering and processingof the SR algorithm may be accomplished in a Field Programmable GateArray (FPGA), a computer, or a computational neural network (CNN)computer. In many cases, the natural motion of the camera producesenough dithering to satisfy the requirements of this method.

The features, functions, and advantages that have been discussed can beachieved independently in various embodiments of the present inventionor may be combined in yet other embodiments further details of which canbe seen with reference to the following description and drawings

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an image of data from an infrared imaging array with awindowed super resolution region of interest;

FIG. 2 is a zoomed image of the windowed super resolution region ofinterest of FIG. 1;

FIG. 3 is a schematic representation of an exemplary focal plane arraywith windowing capability;

FIG. 4A is a schematic representation of the array of FIG. 3 withexemplary readout and supporting processing capability employing a CNNprocessor;

FIG. 4B is a schematic representation of the array of FIG. 3 is analternative processing capability employing a Field Programmable GateArray;

FIG. 5 is a schematic representation of tiled pixels in the array ofFIG. 3;

FIGS. 6A-6C are images of low resolution data, super resolution dataenhanced by sequential operation on dithered low resolution data using afirst super resolution algorithm and a second super resolutionalgorithm;

FIGS. 7A-7C are zoomed images of selected portions of the images ofFIGS. 6A-6C respectively; and,

FIG. 8 is a graph of quantitative comparison between modulationefficiencies for the low resolution input data and the super resolutionoutput data shown in the images of FIGS. 7A-7C.

DETAILED DESCRIPTION

The embodiments described herein provide a “tiled” ROIC architecturethat permits data from user-selected small, two-dimensional imageportions in windowed pixels to be read off of the focal plane array(FPA) at high rates while data from the entire array is processed at aframe rate for lower resolution (LR) data. The image data from theselected windowed pixels is then provided to a near-FPA fieldprogrammable gate array (FPGA) or other processor for evaluation using asuper resolution (SR) processing algorithm. Active mechanical or opticaldithering of the infrared imaging array for the purpose of producingsub-pixel motion is exploited in the processing algorithm to create SRimaging of the selected windowed portions of the array.

FIG. 1 shows an image 2 produced by a 640×512 pixel midwave infrared(MWIR) camera having an FPA and ROIC architecture according to thepresent embodiments. The region defined by box 10 was selected as aregion of interest (ROI) for SR processing, as will be described ingreater detail subsequently. FIG. 2 is a zoomed-in view showing thewindowed region (box 10) in greater detail and demonstrating the SReffect compared to the normal LR data presented on the remainder of thedisplay.

An exemplary imagine array structure which provides for the windowingcapability to access particular ROIs is shown in FIG. 3. While shown asexemplary of an FPA and readout architecture applicable to the method ofthe present embodiment, alternative selectively addressable arrays couldbe employed, The ROIC architecture for the array 12 is composed ofmultiple 32×32 pixel tiles 14 that may be addressed in random fashion.32×32 pixel tiles are used to compose the entire 768×512 array; eightparallel 14-bit analog-to-digital converters (ADCs) 16, shown in FIGS.4A and 4B, are attached to individual tiles so there are never any“lost” digitizing cycles. A processor 18 such as a “cellular neuralnetwork” (CNN) processor shown in FIG. 4A may select any or all tilesfrom the ROIC at the current frame rate based on image analyticfunctions implemented in the processor by a general scheduler 17. Theprocessor, which in an example embodiment is a EUTECUS CNN ProcessorArray available from Eutecus, Inc. 1936 University Ave., Suite 360,Berkeley, Calif. 94704, can provide computing for target detection inclutter, decoy discrimination, multi-target tracking and high-levelfunctions based on that selection ability. A program memory 19 isassociated with the processor.

The native pixels 20 creating the 32×32 pixel tiles for an exampleembodiment are 27×27 um pixels in a 763×512 array providing 393,000pixels.

In addition to addressability of individual tiles by the ADCs 16, 1×1,2×2 and 4×4 “in tile” spatial binning is supported by the ROIC resultingin operating modes that either (a) produce extremely high frame rates or(b) produce very low output data bandwidth for the device. FIG. 5 showsthe general architecture for a 4×4 group of pixels in the array that areinterconnected with symmetrical switches that allow for charge sharingin 1×1 (i.e., complete isolation), 2×2 and 4×4 charge sharing modes. Asshown, the 2×2 charge sharing mode is accomplished by closing the “CA”and “RA” switches 22, 24. Closing all the “CA” 22, “RA” 24 and “RB” 26switches produces a 4×4 pixel charge sharing condition.

Data from the pixels in the selected window for the ROI is read off theFPA at high rates relative to the overall frame rate for FPA as a wholewhich produces the normal low resolution (LR) scene data. In the exampleembodiment, the 32×32 fixed window size is read at 73,000 frames/secondwith the ADCs 16 receiving input from 8 channels of voltage differentialamplifiers 15 for analog video at 10 M pixels/sec.

Active mechanical or optical dithering is imposed on the FPA to producesub-pixel motion consistent with the high rate readout for the windowedpixels. The data received is then operated on using a super resolutionalgorithm such as the method of Projection onto Convex Sets (POCS), theMethod of Irani and Peleg or Method of Total Variation Inpainting, asexamples, which may be implemented in the processor 18 or a separatedigital computing device such as a Field Programmable Gate Array (FPGA)27. As shown in FIG. 4B, the FPGA in certain embodiments may beincorporated in conjunction with the ADC circuitry. Multiple sequentialcalculations on the dithered output provide an enhancement of theapparent number of pixels increasing the resolution for a SR output onthe ROI.

The SR enhancement for the images shown in FIGS. 1 and 2 used the gentleimage motion induced by the vibration of the camera's electriccryocooler to spatially modulate the image data. While not a perfectspatial dither condition, nonetheless qualitative resolution improvementis seen in the figures.

To demonstrate the SR algorithm operation, FIGS. 6A, 6 b and 6C showexemplary images from MWIR image data with a 320×256 pixel @30 um pitchcamera. FIG. 6A shows normal LR data produced by the camera. FIG. 6Bshows the resulting 640×512 pixel equivalent super resolution productobtained by operating on ten (10) sequential frames of the lowresolution dithered input data with the fully-implemented POCS techniqueimplemented in Matlab. CC shows the respective result obtained byoperating on ten (10) sequential frames of the low resolution ditheredinput data with a “Fast”-POCS technique implemented in Matlab. Noticethe Fast-POCS is not quite as sharp and contains less granular noisewhen compared to the original POCS result (FIG. 6B).

FIG. 8 quantitatively compares the three images of FIGS. 6A-6B bylooking at the 8-bit grey levels through a single vertical slice of eachimage, shown as line 28 in FIGS. 7A, 7B and 7C which are enlargements ofthe same portion of FIGS. 6A-6C denoted by boxes 29, 30 and 31respectively. The graph provides a comparison of 8-bit grey level torelative vertical pixel position taken along line 28 for each image.Comparing to the LR input data, trace 34, the Fast-POCS, trace 36, isnot as effective as the original POCS, trace 38, at restoring the highfrequency data shown around the vertical pixel position 30. However, itdoes show effectiveness at slightly lower frequencies, shown betweenvertical pixel positions 50 and 60.

The POCS algorithms rely on an iterative procedure for continuallyrefining the SR result by successively propagating high frequencycorrections onto the SR image. For the exemplary images of FIGS. 6A-6C,the corrections are made pixel by pixel, for all LR images. Thedifference between the original and the fast-POCS methods resides inminimizing repetition by attempting to perform some of the operations onthe entire image, rather than one pixel at a time. This philosophy lendsitself well to implementation on a readout integrated circuit (ROIC) asthose devices can perform massively parallel operations on all pixels.

The windowing capability as described with respect to FIGS. 1-5 aboveallows rapid processing of ROI data within the normal frame rates of theLR processing for the entire array thereby avoiding the requirement forprocessing data from the entire array using the SR algorithm. The SRprocessed data is then inserted into the frame output for the pixels inthe windows of the ROI replacing the LR data normally obtained in theframe for those pixels thereby providing the enhanced image portions asrepresented by box 10 in FIGS. 1 and 2.

Having now described various embodiments of the invention in detail asrequired by the patent statutes, those skilled in the art will recognizemodifications and substitutions to the specific embodiments disclosedherein. Such modifications are within the scope and intent of thepresent invention as defined in the following claims.

1. A method for super resolution enhancement of infrared imaging datacomprising: providing an imaging array having simultaneous, high speed,randomly addressable windows of pixels for small regions of interest(ROI) within a large imaging sensor field of view; retrieving data fromthe ROI window pixels at an increased data rate relative to image framerate; dithering the array to produce sub-pixel motion consistent withthe increased data rate; and, applying a selected super resolution (SR)algorithm to the data retrieved from the ROI window pixels at theincreased data rate.
 2. The method super resolution enhancement ofinfrared imaging data as defined in claim 1 farther comprising:replacing LR frame data for the window pixels with an output from the SRalgorithm for image display.
 3. The method super resolution enhancementof infrared imaging data as defined in claim 1 wherein the dithering isactive mechanical dithering of the array.
 4. The method super resolutionenhancement of infrared imaging data as defined in claim 1 wherein thedithering is optical dithering.
 5. The method super resolutionenhancement of infrared imaging data as defined in claim 1 wherein theSR algorithm is selected from the set of Method of Projection ontoConvex Sets (POCS), Method of Irani and Peleg and, Method of TotalVariation inpainting.
 6. The method super resolution enhancement ofinfrared imaging data as defined in claim 1 wherein the SR algorithm isapplied in a digital computing device.
 7. The method super resolutionenhancement of infrared imaging data as defined in claim 6 wherein thedigital computing device is selected from the set of a FieldProgrammable Gate Array (FPGA), a computer, and a computational neuralnetwork (CNN) computer.